Install the upgraded FME Engine in its own path rather than overwriting the existing FME Engine. This way, you can reverse the upgrade if required.
To upgrade only the FME Engine component when installing an FME Server environment, use a stand-alone FME Engine installer, not the FME Server installer, and follow these steps.
You can get the appropriate FME Engine Installer from this location: http://www.safe.com/support/support-resources/fme-downloads/fme-server/. The web page also includes download links to release and beta installation packages for various platforms.